Multimodal Communication in Computer-Mediated Map Task Scenarios

Multimodal communication involves the co-occurrence of different communicative channels, including speech, eye gaze and facial expressions. The questions addressed in this study are how these modalities correlate and how they are aligned to the discourse structure. The study focuses on a map task scenario whereby participants coordinate a route on a map, while their speech, eye gaze, face and torso are recorded. Results show that eye gaze, facial expression and pauses correlate at certain points in the discourse and that these points can be identified by the speaker’s intentions behind the dialog moves. This study thereby sheds light on multimodal communication in humans and gives guidelines for implementation in animated conversational agents.